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ABSTRACT 



Evaluation practice within state education agencies 
( SEAs ) is*. reviewed fxom a management consultant's perspective. *The 
sfbdy is Based upon a review of literature, discussions with" the 
Northwest Regional Educational [Laboratory Research on Evaluation 
Program, and visits to SEAs .in California, Montana and Washington. ' 
The main findings of the study deal with the current* practice of 
evaluation in SEAs, and with management metaphors which throw light 
^upon the evaluation unit considered as a supplier, of services within 
the SEA, and upon the unit's clients who are its customers for such 
services.. The cenjtral theme of the study is that the basic" paradigm 
underlying SEA evaluation is in the process of change, moving from 
the big. study, advanced methodology glamor* of the late 1960s to an 
o.rganization-based function supporting SEA management in a variety of 
ways, alongside other staff specialists. The study suggests that in 
the long term the SEA evaluation function will condense into two sets 
of roles, namely, that of evaluation systems analysis, and a 
consulting .role. The study then goes on to identify three possible 
intervention strategies: (1) clarification and confirmation of the 
new. paradigm; (2) assistance to the supply side of evaluation 
service's; and (3) the building up of informed and understanding 
demajnd among evaluation clients.. (Author/GR) 
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PREFACE' 



The Research on Evaluation ^Program is ^a Northwest. Regional 
Educational Laboratory project of research, development, testing, 
and training designed to create new evaluaticfr methodologies for 
use in education. This document is 'one of a series *of papers and 
reports produced by program staff, visiting scholars, 'adjunct 
scholars, and project collaborators— all member^' of 'a cooperative 
network of colleagues working on the development of new 
methodologies, * . 

What is the nature of evaluation pracj^ce irt state departments of 
education when viewed by a management! consultant rather than a 
traditional educational researcher? what management perspectives 
might be used to better understand) the- operation of state 
department evaluation units? These and related questions are 
answered in this report by Dr. Jonathan Stanfield, an independent 
management consultant who conducted a reviJkr of the state 
evaluation units in Washington, California, and Montana. This 
report provides an Illustration of a .management consulting review 
study in evaluatioAas well as offering insightful observations 
about the nature of (evaluation practice in state departments. 

Nick L'. Smith, .Editor 
z Paper and Report Series v ' 
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/ SUMMARY 



ThiS^study has reviewed evaluation, practice within 
, state education agencies (SEAs). from a management 
consultant^ perspective, as^part of the NWREL 
Research on Evaluation Program's ongoing enquiry 
into .the utilization and effectiveness of educational 
evaluation. / » •• 

m ' 

The study is based upon a review of literature, 
discussions with the Program, and Visits to SEAs in. 
California, Montana and Washington. 

r * > 

The main findings of .the study deal .with the current 
practice of evaluation in SEAs, and with management 
metaphors which throw light upon the evaluation unit 
considered as a supplier of services- within the SEA, 
and upon the unites clients who are its customers 
for such services . Evaluation i^-not a homogeneous 
function, but in actual .practice embraces at least 
. four separate strands of activity, namely, ^search 
studies, computerized infprmation systems, testing,' 
and expert assistance. « - - 

The central 1 theme of the study id that the basic, 
papadigm underlying SEA evaluation* is in the process 
of change, .moving froA the big study? advanced 
methodology .glamor of the late 1960s to an organ- 
. ization-Tjased function supporting SEA. management* 
in a variety ojf ways,* alongside other staff special- 
ists/ e.g., in budget an<^ finance. From this it is 
possible to show why some of the current issues 

raised by the evaluation community have such a 

i 

* frustrating quality. These include the lack of 

■ . 1 

J . . 



' • . - \ 

-impact of evaliiation on policy, the failure of 
expectations as I to what evaluation can deliver, and 
the inapplicability or prescriptive methodologies 
originating in academe. ' 

.The stud# suggests that in the long term the SEA 
evaluation function will condense into two' sets of 
roles, raamely, that of evaluation systems analysis T 
and a consulting role. This' concept of an, 
evaluation systems analyst may be compared to other . 
applications systems analysts in data processir^, 
e.gy financial, retail,^ engineering systems analysts 
As in all jobs there will be ranges of seniority and 
emphasis. Some will work closely wi|fch 'computers, 
; and others with the design of data collection schemes 
for specific^ evaluations. Some will be entry leve 
positions, and others will be senior technical staf 
Because of their organizational context, the rate and 
extent of such changes depend greatly upon the , * 
management style of the host SEA. f 

« 

The study then goes on to identify three possible 
Program intervention strategies i one dealing with 
clarification and confirmation of the new paradigm 
underlying SEA evaluati^jaf^ng which seeks to assist 
"the si^m ^^sidy oj ^eValuation services, and one which 
is ac ^^3 BR 0 th ® building up of informed and 
fl und er^gaS^t^d emand among evaluation clients. 
Recomm^^^^s are given for program activities 
in each Qf^these 1 categories . 
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1*. INTRODUCTION 



This is the final "report" on ~ the study-i Management 
Revlew^o f Evaluation Practice , carried., out by the^ 
consultant for the Northwest Regional Educational 
Laboratory (NWREL) under the direction of Dr Nick 
Smith, •Director, Research on Evaluation Program. 



/ 
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The study reviews evaluation practice within state 
education agencies (SEAs) from a management, 
consultant's perspective. Accordingly, this re^brt 
is written primarily as "a .consulting, report tcL^f 
client (the Program), rather than as a research paper, 
addressed to an audience of scientific peers. 

The study is^related to -two J areas of recent Program, 
activity. The first is 'an enquiry into SEA evaluation 
practice, which seeks 1 to document what SEA evaluators ' 
really dp and the nature of the settings in which they 
work. Thus, Smith (1980) concludes by suggesting 
that more attention sho.uld be paid to , understanding 
how evaluation functions within organizational and 
social ^contexts rather than to questions of causal 
modeling and' experimental design. ^The second area of 
Program activity is that concerned With "the metaphor 
series of papers, in which* potential new approaches 
to evaluation are explored as metaphors derived from 



practices in other fields, e.g., operations research, ' 
investigative journalism. % 

■ , - • • * 
Originally, the study was* intended to identify a range 
of candidate metaphors from the field of management, 
interpreted -broadly, to assess their applicability to 
SEA evaluation practice, and to 'derive conclusions 

» * ' 

V 

, - 1 - . • 

; ... a 



-from such, application. Thi? proved to be somewhat 
too simple 'an approach flue tn thft oomplax and varied 
nature of SEA evaluation practice, which renders most 
metaphors either superficial or of only limited 
relevance. Thus, the study 1 s ^results are derived 
mainly from a direct analysis of SEA evaluation^ 
activities, rather than through the indirect route of 
metaphor'-. 



The study is based upon two major inputs 

- a review of literature on SEA .evaluation 
practice, largely furnished by the Program, T 

• as a means to developing an initial approach 
to "'the subject? and 

* 

- - visits to a nWber of SEA evaluation units, . 
^ as a means *b6 understanding something o£ tlie 
richness ancl complexity of what actually 
happens in practice. 



The main text of the report following has three 
sections, dealing withj ? 



th^ primary findings of thg 'study (not all 
original by any means )i 



- discussion of - the central them^ of H:he 
'study, some current issues, long terjjo, trends , 
and types qX intervention strategy open to 
the Program j artd ^ ^ 

recommendations for further action. 
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There, are a number of appendices where some of the 

material is treated in more detail*. - ■ ' 

-The consultant would like to thank the Program 
Director, Dr Nick Smith, for his assistance in the * < 

course of th*e study, not only in matters of overall • V 

direction, but also discussion of the planning, 
/ ^.nierim and draft final reports . ' 

1 

■ ' The views expressed in thrs* report, however, remain 

those of the consultant, and are not necessarily those, f 
of ^the ^Program. 
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0 2*. FINDINGS " .' ' \r> 
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Major findings^^of^the studjr are preset-fed below. - „ . 
They are not all necessarily original; but are - 
presented here for their significance tdfco the studyT" • 
and to provide s^jme- introductory basis for the • 
discussion in the- next section. T-hey f all ^nto fo.ur 
groups i v 

- "evaluation in an SEA" context . that is, 

direct findings on. the nature of SEA 

- ( 



Nsye^aluatictfi itself? . 



- management metaphors t that is, those 
/ activities within organizations whi^h 

appear .to. be of some relevance to 
' understanding SEA evaluation practice; 

6 - management consulting and evaluation , 
that is,\the relevance of management 
consulting* to evaluation practice; and 

- issues' gf, concern , jthat is, some of' the 

y . ^ r 

issues that surface repeatedly in the 
literature and were .confirmed by the site 
-visits carried out by the consultant to 
individual SfiAs . 



2.1 Evaluation in the SEA Contextr 



v 



c 



Findings on the nature of evaluation in the SEA 
, context, relevant , to the present study, are given 

overleaf. '(They are discussed more fully in Appendix A.) 



■J 



•A 



(a) SEA 'Evaluation is an - organizational function, \ 
carried out oy an organizational unit loca'ted in 
• an organizational structure. It therefore differs 
from the view implicit *dn much of the writing jon 
•'evaluation, i.e., a single, one time study, 
' organizationally independent. 

* 1 \ / 

(bj* The SEA' host organization is a distinctive kind 
, of organization. It cannot readily be compared 
to private business fbr the obvious public - 
•private reasons. Nor does it resemble other' 
government agencies closely due to the open, 
political .nature of i'ts activities , - and the 
atypical 'emphasis upon outcomes that other agencies 
do not share. It is this emphasis upon outcomes 
that gives one of ""The primary justifications for 
% evaluation in SEAs, as presently conceived. ■ 

* * 
\(c) SEA evaluatibri is new. It dates ^ in its present 
form, from the *1965 Elementary and Secondary 
Education Act, which mandated evaluation for the 
new federally funded compensatory programs. 

**. (d) SEA evaluation is a diverse activity, varying in 
..7 title/scope size of unit, organizational location, 
* t and practice with regard 'to contracting work out 
'or carrying it out inhouse. 

le) SEA evaluation is not a* single ( homogeneous 
activity,. Four -major types of activity are 
practised under the general heading of evaluation 
(although only the first -o^tegpse may be^ regarded 
as evaluation by thg- purljra^r 



) 
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i - ad hoc, one of a kind research analyses 

of issues and programs, either for 
# ~ retrospective program assessment or for 
* prospective policy fbrma'tionj * 

: ■ . ■ • . 

- inf ormatioiv systems^ development and 
operation, containing data on schools, 

their operations, students., their. 

/ " \. - • * J J 

' performance, etc-, j . * x 



testing, or, more generally, routine 
obtaining of information *on a population 1 N 
oi interest through some instrument of 
enquiry, e.g., a questionnaire? and , 

« 

- expert and technical assistance to LEAs, 
other SEA staff, legislative committee 
staff, etc% While this may consume a 
great deal of staff time in small bits , 
it is a general function of staff specialist 
groups in all organizations and is not 
specific to evaluation. 

(f ) SEA evaluation units are not ,the only units in 

SEAs that carry out evaluations. Some programs 

v 

carry out their own, with or without the help of 
the evaluation unit. 



(g) SEAs include- other units whose work relates closely 
to i 4 needs to be coordinated, with , or overlaps, that 
of the evaluation unit. The se* include units with 
such£names as management information center, 
-^financial audit, policy and planning, etc. 



2*2 Management Metaphor^ * '„ * v 

There are. two sets of management metaphors of ^potential 
rel^vasice to SEA evaluation.' The first relates to 
the SEA evaluation uniVs activities-* themselves - 
■ the unit plays the role* Of a supplier of services. 
The second relates to the SEA itselrf^and the unit's 
clients, which play the, role of customers for the * 
unit's services. We shall call .these supply * 
metaphors aiid demand metaphors respectively, (They 
* are discussed more fully in'Appendix b\ ) 

The Program's metaphor .series of papers has identified 
a large number of 9 potential supply metaphors. 
However j it is * important to extend consideration to 
demand metaghprs in order^ to do justica to the fact 
that evaluation is neither self-sufficient nor an end 
in itself, but increasingly shaped 'by Ijjie character 
of demand in its immediate environment. ■/ 

* L 

2.2.1 Supply metaphors / 

s . * " 

One oVShe difficulties of identifying metaphors for 

^SEA evaluation is that it encompasses a number of 

. distinct activities , as already described . Since. a 

single metaphor is therefore unsatisfactory, the 

consultant,, has sought rtietaphor(s) for each of the 

component activities. Although all four activities 

are represented below for completeness, metaphors for 

the last two turn out to be somewhat trivial. As can 

be seen, to jifistinguish metaphors from the activities 

themselves, different names ire used. Mostly^ the 

metaphors view evaluation a&. an activity of staff . 
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specialists, not'ag an activity within the management 
mainstream of the organization,- that, is, of . the SEA. 

(al) Operations Research/Management Science (OR/MS > . 
$his is .concerned with ad' hqc analyses to support 
management decision making, * based upon appropriate 
theoretical frameworks . % At the technique level 
it is. usually based upon mathematical, financial 
and physical science models, rather . than # upon 
behavioral science, ^bujfc 'from a* management point 
of view thys is not an ^impofrtant distinction. 

(a2) Marketing/Market Research . if second .metaphor 
for the research' analyses Element of. evaluation 
is marketing and .market research , which are- 
traditional business funbtions . Recently they 
i have been seen as increasingly relevant to 
government ^operations . \ln the present case, 
for example-, they deal with needs, assessment, 
market segments and segmentation, the success 1$ 
of products and services -as consumed by 
customers, in fact, the whole machinery of market 
feedback. This is a major part of what evaluation 
is all about. They arje also much more co#£erned 
with human behavior than OR/MS . 

(b) Data Processing/Management Information Systems 

^DP/MIS) > This is concerned with thfe development 
ind operation of information systems to support 
/management on a planned- or routine basis r both, 
through on-demand enquiry as well-as scheduled 
repgrts, and may include a variety of analytical 
tools ♦ ^ \ 

/ 

■ •' v 16 ■ 
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(c) Surveys .' This is 0 one erned* with the periodic 
, survey of a population of interest, e.g., as in 

market research' or .public opinion polling; 

(d) Consulting. This may be internal or external 
consulting, and extend* from a casual telephone 
enquiry to a^mrni project. * 



2 .2 . 2 Demand metaphors 



The two main metaphor§.'fere have * to do . with the 
varieties of m^nageria^ decision making on the one 
ftand; and the .necessity; of information feedback from 
the field on the other./ These are, of course, 
related^ but 4:he metaphors and their associated 
literatures have different emphases., and draw 
attention^ to different features of the v evaluation 
environment . 



(a^) Decision making categories . For many years the 
management community has categorized decision 
.making into three categories, concerned with* 

• - operational control j 

• * 

- management control i and 

- strategic planning* 

TITe distinction between these categories is* 
sometimes blurred, and- associated roles and 
responsibilities may overlap. A useful indicator 
of the level of decision making is the, relation to 

- 9 - 
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the budget." ^ISf&rtagem.ent' control is concerned with 
setting and x mee ting anni^^^ai^a^ budgets, 
. - operational control is concerned with^^i^^iwal 

tasks that the budgets fund, and strategic planning*- 
is concerned with coherent views- of the future 
- extending over several budgetary periods.* , These 
categories have been used mostly in business 
settings* In government, strategic planning can 

often be equated with broad policy formation. 

) * * ^ 

The information required, for each category is ' 
'different, and consequently different' means of 
* supporting each category have arisen. In the 

K 

pi^sent case, operational control is mostly to 
be foundffF~^ school level. 

Evaluation will support eaahT^rpe^oiL^decision N 
. making differently. - " * 

' • -I 

(b) Feedback control . Feedback control systems ^gtre 
mo f dels for a wide variety of biological and social 
processes, as well as underlying iuhe industry of 
engineering control from thermosxats to missile 
guidance. One of ilian's distinctive attributes 
is his ability to learn from the outcomes of his 
actions. If he canriot observe these outcomes, 
because of extraneous factors, or if the rates of 
change in the environment occur faster than he can 
detect, then feedback is .impaired; and his activities 
^re less likely to. be reliably effective . 

(Phe saiflte is true for SEAs. If they cannot track 
what is happening in the field in an accurate and 
. timely manner, then their ability to direct the / 
state educational ehterprise is seriously impaired. 
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Irhis metaphor is/ closely related to the preceding 
one, but has an; emphasis upon* information, speed 
- of response, and noise, that, is particularly 
appropriate to the function of evaluation *in an 
SEA. ' • *• 

> 

*• i r , 

k » ' • 

2U3 Management Consulting and Evaluation 

i 

One of the Ingram's izrterests is the extent to which- 
management consulting itself migh^ be a metaphor for 
evaluation. As identified earlier, there is clearly 
a part of- SEA evaluation practice that is a consulting 
activity . It therefore shares with management 
consulting a common concern with what might be called 
consulting hygiene, that is, good client relationships, 
timely reporting, etc. .The. consulting activities of 
NWREL also share this concern.* 

-^In the view of the consultant, the consulting "component 
o^NSEA evaluation is* mostly a fprm of technical x 
consulting rathei^ than of management consulting. TV 
is mainly due to $he fact that evaluators are n<St 
usually 'asked questions for which the management 
consulting approach is the most appropriate. 
'Technical consulting' is narrower, more focussed, 
concerned with the supply side, and "how to" questions. 
Management consulting is broader, concerned with 
demand and often with "what for" questions. Although 
there can be a degree of overlap between the two, 
nevertheless, the two types of consulting acquire 
different styles,, attract different stdff , serve 

> ' % clients differently, and generally operate in 

different markets. 1 Management consulting is discussed 
further in Appendix C. 
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A Issues of Concern e 

SEA evaluation today is characterized by a number of 
issues that surface repeatedly, both in the literature^ 
and also in discussions with individual S$As',. These 
issues are often only* t touched upon partially, but w / 

when all the references axe taken together, a picture 

f - * " .' 

emerges of real issues that -need to be explored, ' 
understood, arid responded to, if the future of SEA 
evaluation is to match its^ early hopes/ Three 
issues are discussed more fully in the next section, 
' namelyi " , 

( 

- evaluation does not have. the impact on 
- • SEA^njanagement that its proponents believe 
it should; " 



- expectations of what evaluation as a ' 
technique can deliver are not met (and 
this is sometimes true of expectations 
of , the educational process itself) \ and 

-» there is s v ome degree of tension between 
methodologists and practitioners. 



20 
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-3. DISCUSSION 

In this section we discuss a number of topics that we 
believe are important to an understanding of SEA 
evaluation today and to its future. These include 
the three issues mentioned at the end of the last 
section, as well as long term trends in SEA evaluation, 
and types of intervention strategy open to the Program. 

In considering these topics, the consultant has come to 
* hold the view^ that the dominant paradigm underlying 
SEA evaluation is in the process of changing 
significant^ • This proposition will be discussed 
first, since it is so fundamental that it sheds light 
on most of the other issues, as well as giving an 
orientation towards the future, and to possible 
intervention strategies open to the- Program, 



3*1 SEA Evaluation i £ Paradigm in Transition 



Much evaluation literature treats evaluation as 
synonomous with large, one of a kind social research 
studies, carried out by independent contractors, 
utilizing advanced methodologies, highly influential, 
and lasting manj< months or even several years. This 
is not characteristic of 'SEA evaluation^ 

Let us go back to 1965 » and consider both the mission 
and the methodologies of the new federai programs in 
compensatory education. On the mission side they - 
were seen as & means of^ idealistic social change and 
not just as education, part of -the administration's , 
Great Society program. The poor and disadvantaged 
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would be helped in a demonstrable way. This led to 
an emphasis on outcomes that other^government spending 
does not share, e.g., roads, health, defense. On.* 
the methodology side, 'Robert McNamajra was introducing 
PPBS, and the climate favored a methodological 
rationalism. The climate has since changed. Thus 
evaluation acquired early 4 on two set beliefs i 
compensatory education will really change outcomes, 
and this change in outcomes can be shown by an 
appropriate methodology. . 

The first evaluations were large, one of a-kind studies, 
performed by independent contractors, from academe as 
we/11 as business, who considered themselves major 
contributors to policy through their research - they 
were the peers of the policymakers. 

SEA evaluation, influenced-; by the mandatpry requirements 
of programs such as Titje I, tended to form" itself in 
the mold of 4 such early federal activity. (Of course, 
some SEAs were too small to do very much in this 
direction.) However, the* realities of SEA: evaluation 
are mostly quite diXferezit, in terms of organizational / 
context, size and skills available, timescales, funding 
l^vel, and so on. The SEA evaluation unit., in fact, 
plays a SEA management §upp<art role, and this role will' 
develop in different way^ depending upon the host SEA's 
legislative context, size, leadership, organization, 
and allocation of responsibility between the variety 
-of specialist staff units that serve its management, 
including evaluation. • • 

The paradigm of a field of activity is the most basic 
underlying concept *of what it is £bout. It is an 
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organizing concept, which brings to mind certain 
characteristic questions and approaches*.- Indirectly, 
it directs research f and development; 1^ p£o|ri.des a . * 
common basis for integrating the efforts of different 
parties. The paradigm underlying SEA evaluation is 
in the process of changing from the autonomous conduct 
of a bigj^tudy to the 0peration of a -management support 

unit in an organization. ' S 

*** 

\ 

Otit of date paradigms lead to a widespread sense of 
confusion and frustration in practice. That this is 
so can be seen by considering some of the current 
'ipsues of the evaluation community, which are discussed 
below. However, it should be noted that updating the 
paradigm will not by itself solve all of the problems 
faciAg^^e valuation in SEAs. * There will remain difficult 
tasks, except that they can now be approached directly 
and pragmatically, without the confusing influence pf 
an out of date paradigm. 



3*2 Issues in 




uation 



We now discuss three ^mainstreams issues of the SEA 
evaluation community in the light of the above. 

(a) Impact . There- is no a priori reason why- 
evaluations should have the major, much less the 
• determining, impact on policymakers that the 
literature so often implies. This notion arises 
out of the original paradigm underlying SEA 
evaluation, in which the social researcher played 
an influential role in the development of federal 
policy towards both programs and evaluations . 
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Consequently, when writers speak of lack of impact, 
they are harking batck^to'a golden age of evaluation 
(which still exists, to some extent, in current 
*major federal evaluations) • In the same vein, 
they imply a degree of autonomy on the part of 
the evaluator-researcher that is appropriate to 
that time. Since SEA evaluation is a specialist 
staff function, such autonomy is not appropriate 
•today in this setting. There can be creative 
initiative on the^part of- the unit, but authority 
and sponsorship to carry out studies, gather data, 
develop systems, etc., should come from the clients 
served. This isu*lear from the OR/MS and DP/MIS 
metaphors. 

Consideration of the type of decision making that 
evaluation % supports sheds further light on tire 
question of impact. Most evaluation impact is 
assumed to be on policymaking, whiGh corresponds 
to the broad area of* strategic plannirig in business 
Thi^is the area where th$ inputs range the most 
widely, may well be qualitative rather than quant- 
itative, approximate rather than exact * Certain to 
the future rather than to thejo ast oj ^ present, etc. 
In short, evaluation input is only one of . a number 
of inputs, and may well rank lower than others of 
a political or financial nature. 

The situation is compounded by technical limit- 
ations. In many cases, evaluations produce "soft" 
data which can be successfully challenged, for 
example, j.n the adversarial setting of legi^±«rtj,ve 
hearings, or*yeven in the courts. 



/ « 
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In contrast; routine inputs" to prog^ain management 
decision making from a* database with which «pae 
familiarity has been gained in the* past .may >ell ^ 
produce rapid and immediate action* as, 'for 
example, when sales figures drop be\ow some 
threshold in business. ^ 

Impact is a lso af f ecte£ bjr A the quality of 
evaluator-client communication It appears th^ ' 
although lip service is paj^a to this, in most case^ 
there is room for considerable improvement. In 
general, people habitually underestimate the amount 
of time and effort that must go into communicating 
(needs, capabilities, results, etc.). Consulting 
experience brings this- home.*' 

0 

/ • 

(b) Expectations *. There is frequently a gap between 
what evaluation promises, or is perceived to 
promise, and what i,t can deliver. This, too,* 
can be traced back to' the first Utopian days 
of evaluation, when major studies of national 
significance wer.e commissioned by the federal '« 
government (and as some still are). Since then 
SEA evaluation tends to have been oversold, both 
by evaluator*s wishing to maintain their influence, 
and by clients wishing to get programs funded or , 
legislatioh passed, and incorporating promises. of 
evaluation without being completely clear that the 
results promised are in fact deliverable. 

With the change of context from the federal to the 
% state arena, a change <6f timescale has also taken 
place. State level clients y/aht results fast. 
This leads to reluctance on the part of elvaluators 
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to reduce qlient expectations, and 'there ig 



• - consequently frustration when they -^re not met. ^ ; 

^However, it is likely that certain things just . , , ; 
cartfiot be g^und out in a hurry. ' Compare, for ' % / 

example > dieting or drug research. . It fcay take 

* years* to prove results and causal links. tfhere 
are inherent limits to what can b^ achieved and* 
how fast. These comprise the- current state of 
the art and should be openly acknowledged t nothing 
is gained by pretending that these limits # do not 
exist * 

(c\ Methodologtetfc and^prastitionefs . Evaluation is 
^ Nt young- discipline and 'bears all the hallmarks . of - \. 
this condition. ' Typically, there is an incomplete 
separation of theory and practise, as, for example, 
also occurred in information science, arti^fioial 
* > inlJSliigence aijd machine transitions In all „ of 

* these, utopianXclaim^were .made for the^iheoriea * 
, by ac^pmics, , and their immediate* prescriptive 

application wa^ urged. ; This- immediate application 

was a failures Recrimination* and pessimism 

followed . - ^ 



Some years la^er'\he situation has changed. 
Academics pursue^ theoretical questions in ; one 
reward setting. • Practitioners, pursue practical 
'ft questions in another. Both represent cumulative 
- *V knowledge bases, linked by examples of problems 
- flowing from practice, to academe, and by results 
^from theory flowing in the opposite direction. 

. d-' „ 

* In the" consultant's visw, evaluation is just 
coming to this "point of separation* and there is 
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** understandably some reluctance on the. part of the 
people involved, to lose touch with "the other side" 
However, It will probably be better 1 for all 
concerned when 'the two groups have distinct 
identities and a degree of mutual respect that 
sometimes appears to be lacking at the present. 
Such a separation represents both, a responsibility 
and an opportunity for SEA evaluates ^fco develop- a 
body of ^professional expertise appropriate" to their 
setting. 



3 Trends in SEA 'Evaluation 

To predict the # future of SEA evaluation is .highly 9 
speculative, but from the perspective developed in 
this study' possible directions of development? can be 
suggested. . ' ' 

- * * " » 

The role of evaluation in- supporting SEA long .range , ' * 
or strategic, planning* will decline since this is 
likely to remain a 'permanently underdeveloped client" 
, area . * ' ' SEAi suffer such abrupt .changes of social- 
- priority and funding. levels that any strategic posture 
can be obsoleted almost overnight, and consequently 
SEA management tends to be pragmatic in approach, not 
often, given" to planning beyond the next budget. ' 

Evaluation will therefore lose one. of its earlier 
. claims to glamor. As the new paradigm becomes more 
accepted, it will become clear that evaluation is one 
of a number of support functions available to SEA 
management, ^Evaluation will probably remain the 
internal SEA authority on fiefd data collection. 



This role in understanding and designing feedback 
^ system* represents an important contribution to SEA , 
management that is not presently -duplicated by other 
SEA staff units , and consequently may represent a 
stable long term role for the evaluation unit. 

Costs arid response time will cause the reduction 9f 
major research studies to a minimum,- except for those 
' needed by existing or proposed categorical programs*. 
With the advent of block grants there will probably 
be less ^of .these . ^ 

p * 

Costs and response time v^Lll also favor ; the growth 
of the DP/MIS component of -evaluation activity at the 
expense of the research study component. This mirrors 
the f way in which DP/MIS has** come to dominate OR/MS in m 
organizational influence and investment. The growth 

• of the DP/MIS component may initially be masked by its < 
greater efficiency. This irony comes about because 
although it may be dbijig more and mor,e, '"once .systems 
are set up and running, ' costs are small, and even fall 
along with falling hardware costs. A secondary effect 
of the growth o£ operational- systems, however, is 
-"maintenance^ drag" which affects both software and % 

% procedures^, andvtfhlcfi further reduces resources for 
major discretionary studies.. 

The actual organizational role o*f evaluation will 
depend upon -individual SEA management Some units will 
remain^independent, others be absorbed by program 

.management, others merged into an expanded DP 
operation, •and yet others take over the department's 

^DP activities • 
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While differences will always remain, not least .because 
-of size, the consultant believes that evaluation 
activities will eventually condense into two clusters; 
*n evaluation systems analyst role working closely 
with integrated database systems (sometimes engaging 
in major studies or. data collection design projects), 
and a consulting^role, both internally and for LEAs ~ 
and legislative clients. 

To expand upon the first of these roles, an . evaluation 
systems analyst may be compared to other applications 
systems analysts in data processing, e.g., financial, 
retail, engineering systems analysts. The results 
of their- work is to cause a computer system to support 
some real world activity. They know enough about 1;he v 
activity itself to be able to think in terms of the 
system's impact on it, and enough about computing to 
understand how ,to structure the data .processing 
requirements for effective implementation. They are 
concerned with what inforojation goes into the system, 
where it comes from, what databases are maintained, 
and what- reporting or enquiry facilities are needed. 

As in all jobs, there will be a range of senior iy^^"" 
and -emphasis. Some will work closely with computers, 
other£ with the design of data collection schemes, for 
specifics evaluations. Some will be entry level 
positions, and others will be senior technical staff.. 

Staffing will reflect the growing maturity of the 
field . First generation staff, the staff of the first 
ten years, typically have a variety of backgrounds. 
Second generation staff will have college backgrounds ' 
in evaluation. However, be causae of the likelihood of 




a growing separation^ir^-tyeen academe and practice; 
college trained evaluators may take some time to settle 
in to the practitioners 1 world . 

A more specific problem with staffing is related to ' 
-the fact that 5 evaluation has the four .components 
mentioned above. The personal s^lls and temperament 
required for each are different, ranging from precision 
and methodological sophistication in the case of 
research analyse^, to the more pragmatic, results- 
oriented appr^afch of expert assistance. In some 
cases, this may lead to tension between the quest for 
technical excellence s and the desyre ^to* provide effective 
help promptly/ In large S>EAs , Ifiis may be alleviated 
by the fact. that staff numbers will permit some degree 
of specialization.* ' In small SEAs ,jf however , a degree 
of versatility will be required. If the^jti^ds 
outlined above da take place, then thei^consideratioris 
will be less relevant, ^ since ' the skills .and temperament* 
required for systems analysis and consulting are -more 
similar . 



3*^ Intervention' -by the Pro ay am : 

t 

In the light' of the above, the question arises as to 
what the Program can do to bring about constructive* 
developments in the S3A evaluation field. There are 
at least three different types of strategy, and each, 
leads to a number of recommendations in the next 
section. 

(a) Clarification . Work to clarify the picture* 
proposed here of a. new paradigm underlying' SEA 
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evaluation, i;e., that#>of a managements support 
function, so that itJcan provide a realistic 
basis for cooperati/e activities by' all parties 
concerned, 

'(b) Supply side .help * Work to build up the 
capability of the evaluation unit to offer . 
> services to the SEA, through tools, techniques, 
meetings, etc. * " J 

« * 
(c) Demand side helr^ Work to- develop an under stand - 
ii^g among SEA clients and top management of the 
role and value' of evaluation in leading and 
managing an SEA today. This- may seem somewhat 
out of scope, but the concept of demand-pull can 
-be; as important" in developing services as the more 
traditional supply-push. The support of the state* 
superintendent of education will do more for the 
rol^n of evaluation in^ an SEA than the latest 
techniques . 



4. RECOMMENDATIONS t 

Recommendations to the Program arising from the study 
-are given below, divided into three groups, reflecting 
. , the different intervention strategies outlined at the 
j end of the last section. 



4.1 Clarification 

(a) Clarify and validate the proposition} that the basic 
paradigm underlying SEA evaluation is changing, 
along the lines outlined above , and work towards 
it6 acceptance by the various parties 'involved * 

"(b) Related to this, develop a model of SEA management 
and function, differentiated into, say, thr.ee size 
groups. This would include policy and program ^~ 
development, decision making, legislative and 
executive relations, budget -preparation, and key 
information flows. While reasonably comprehensive 
in general, the model would fopus on developing a 
• representation of the role of evaluation in SfeA 
affairs . 

(c) Develop from these a number of detailed c^e 
studies reflecting a range of practical 
evaluation requirements i^rea^irfeic environments. 
These could serve as the basis f or Vui 
research in academe, for sharing approaches "between 
practitioners, and for discussing information needs 
with clients. They would' be more detailed than \ 
some of the case, studies appearing in the literature 
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^.2 Supply Side Help 



T 



\ 



(a) Extend the work of the Research on Evaluation 
Program into a more derailed review 4 of the scope 
of* SEA evaluation units and* their activities, » 
trends, job descriptions, inter-unit relations, e^c 

(b) Develop a taxonomy of evaluation methodologies 
organized by such attributes as response time, 
accuracy, cost, respondent burden, etc. Be 

- realistic on any limitations . Identify profile 
♦of desired methodologies, i.e., gaps in the 
taxonomy, and commission research to fill the 

most important; gaps . 

T 

(c) Consider a metaphor study to explore,* the marketing/ 
market research metaphor further, as a resource 
service for SEA evaluation units, rather th^n as a 
'means of extending methodology* 

■ 

/ 

(d) Garry out a study to examine the information 
systems that SEA units are developing with 
attention to /capability, cost, hardware, 
transferability, etc . / 



(e) Organize peer group meetings for evaluators on 
* specif ic topics to share experience. These would 
be working meetings. , A newsletter, -occasional 
journal, or information exchange might also be 
organized. 
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^.3 Demand Side Help — K • 

(a) Develop detailed life cycle studies N of policy 
and program development in order to identify the 
potential role of evaluation, as a means to helping 
clients perceive this role. 

(b) Carry out ?ome kind of market research enquiry 
into clients 1 needs for evaluation, perhaps using 
the Delphi technique. 

(c) Organize peer group meetings for program managers 
and top policymakers,, even state superintendents 
(different meetings), addressed by prestigious 
experts on information and management, e.g'. , Peter 
Drucker, These would require careful preparation, 
choice of location,' and personal invitation from 
the Director pf NWREL. 
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A. EVALUATION PRACTICE IN SEAS 

This appendix describes evaluation practice in SEAs 
as the consultant has come to understand it in the , 
course of the study. It is based mainly °upon v 
materials provided by NWREL, q.s well as visits to a 
number of SEAs, and will probably contain little new 
to those familiar with the subject. 

This description is selective, reflecting ,the special 
interests of the study. That descriptions of 
evaluation are frequently, perhaps inevitably, ■ 
selective has been amply documented by the metaphor 
Series of reports sponsored by NWREL during the last 
few years, e.g., Guba (1979) * Page (1979) The 

V 

present study is one more example of how the conceptual 
framework and analytical tools of the reporter influence 
what is perceived as significant in the field, and how 

* it is described and structured. (What is -sought 
ideally, of course, is an approach to evaluation which 
has substantial, aryi lasting, i explanatory power and 
the .ability to suggest and support constructive 

• strategies in the field. 

SEAs ara organizations intermediate in the public 
•education hierarchy, between the federal government, 
acting through the US Department of Education, and 
local districts. They have relationships -with each 
of these levels , as well as with their own state 
legislatures at state level. These relationships 
may be summarized as follows i . 1 

- federal relationships \ SEAs are "responsible 
to xhe USDE for the funds flowing into the 
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' state for specific programs-, e.g., Title Tj 

- state relationships : SEAs are responsible' 
to the state legislature, often through an 
intermediary state school board, for 
managing the state educational enterprise, 

k to the extent that the SEA has an effective - 
leadership and/or control function in the 
state; and 

- local district relationships i SEAs have an 
oversight role, often on behalf of state 
and federal funding sources, to see that 
funds are spent; as intended and education 
is carried- out to appropriate standards, * 
as well as a technical assistance role and 
a policy or guidance role . 

None of these, relationships, however, are strictly 
executive, in the sense of one level having the power 
to control lower levels by direction, but there are 
implicit sanctions through funding that can be 
strongly influential. It should be noted that 
although the federal relationship is highly visible 
and has Influenced evaluation deeply, only a relatively 
small amjpunt of funds for education actually come 
from federal sources. ' * 

For all practical purposes, evaluation in SEAs in its 
present form_ dates- from 1965 and the passing of the 
El ementafy- and Secondary Education Act (ESEA). This 
increased significantly the amount of money flowing 
into states from federal sources for education, and 
brought with it^ao^e than corresponding requirements 




for, evaluation ^md accountability or monitoring. 
"On tfre~one hana , the" worW~bT~ the Tederal^prd^ams" " 
had to be established in relation to the national 
policy arena, and on the other* hand, the money had 
to be shown to be spent appropriately to the intent 
of the program planners. In the language' of 
evaluation, outcomes and process were both to be 
evaluated . 

> 

It should^ be noted that 196 5 was also the year in 
which Robert McNamara's Programming, Planning and 
Budgeting System (PPBS), originally instituted in the 
Department of Defense, was promoted more widely H| an 
aid to effective public management in general. 
Evaluation, as originally conceived, and PPBS shared 
the same .Utopian rationalist expectations, which many 
now believe to have been inappropriate . The mood of 
the times has- changed. „ 

Most^of the states now have evaluation units or 
departments with specific staff designated as 
evaluators. However, there is a great difference 
between states in such ^matters as* 

4 

-Nthe title of the unit - some combination 

. - oi research, testing, e|ptluation, planning, 

Accountability may be used; 

- the size of the unit, which tnajfc range from 
0 to 70+ j and 

- contracting practice, ranging from mostly 
inside to mostly outside. ~' 
(Caulley and Smith, 1978) ' - 
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Perhaps size is the most important factor relating 
to competence, and it is noJ^siirprising that this 
should vary considerably, from SEAs .that are. 
reluctantly forced by the federal requirements to 
some minimal level of evaluation, ' to those that pride 
themselves on greater sophistication than the federal 
guidelines* This range of skill has been described' 
in Boruch and Cordray (1980, pp3-13,l>) under the 
headings of exemplary, compliance-oriented and 
emergent SEAs. v 

Evaluation units perform services ftfr a number of 
different clients, for a variety of purposes i 



federal government i these evaluations are 
required by law, and are often specified in 
some detail. They are intended to provide- 
the federal government with information on 
the funded programs from a variety of points 
of view, e.g., descriptive., summative, 
outcomes/impacts ; 

*, 

state legislature^ may direct evaluations 

to be performed both to inform policy 

making .activities and 1k> monitor new ) 

programs, 1 sometimes going direct to SEA 
» 

evaluation unit staff, without going through 
the SEA administration j 



Si 
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SEA top management t bpijh in response to 
external requirements and in response to 
its own 'autonomous development of! policies 
and initiatives, requires the support of 
the evaluation * unit in providing specific 
infortaation and- analyses j 
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- SEA^^oCT^managers : t hese require v.... 

' *■ evaluation assistance to discharge theirf 

, — s t ^ 

program management responsibilities, both 
. within their own offices, and in order to 
show^ funding agencies (both state and 
federal) how the money is spent ? and 

- local districts : these require technical * 
assistance r in carrying out local Evaluations , 
both for their own purposes, as well as to 
meet requirements arising .from state of %t 

or federal funding. x 

Although these appear to be neat and clean distinctions, 
visits to actual SEAs showed the consultant that SEAs 
are complex organizations without strong organizational 
boundaries, and evaluation units are caught up in a 
variety of client^ relationships which "can* exert ~ % 
extremely taxing pressure on^aluaflon st^ff . 

Turning to what evaluators actually do, evaluation is 
sometimes treated as a monolithic, 'indivisible activity. 
This is ndt the case in practice, and a number of 
distinct activities are performed in providing the 
services above. This fafit makes a single definition 
of evaluation, such as many, writers propose, a mis- 
leading 'goal, since such definitions t tend to reflect 
only a subset of the field of evaluation practice. 
These activities includes , . - 

f~*-> research analyses i such as special analyses, 
enquiries studies , which 'entail some study 
design, data collection, analysis and 
reporting. .These are mainly o/e of a kind 
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projects -conducted on^an.ad hoc basis* 
serving either some high, level* prospective • 
activity,- e .g. , policy aojd planning, pr 
some • previously mandated requirement for 
retrospective evaluation; 

- inf orniation. systehi activities i such as *■ 
the development of. systems giving information 

. on school operations, test scores, etc., 
that can be used to generate periodic 
reports, as well as respond to on-demand* 
questions) • * > % 

* 

testing r,th'e Actual management of a testing*, 
program, from requirements to actual results \ 

£nd <* ' . - * • ^ 



- expert assistance i 3T-or example, ^avice to * 

> local districts on tesUjLng, testimony to a * 
legislative .committee, inputs' to SEA pblicy 

- ^development, eta.. i i 



As in the case of types of . client^abbve these may be 
somewhat, too exact to be used without some' qualification 
Nevertheless, they do seem to span the range of activ- 
ities, that evaluation units perfotm, .and are used as 
indiGatqrs of that range^in the course of -the study. 
In practice, as might; be expected, there is some 
overlap .between, them ♦ For example, testing may 
result in data being fed into an information system, 
or expert assistance may involve advice on a research 
analysis for an .SEA colleague. 
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In the evaluation literature a number of issues are 
currently being discussed. These include; 

- whether the dominant realities of evaluation 
are political or technical. - In the former 
point of view, evaluation is intimately part 

t of the political process and its success 

will be partly political. In the latter 
* case, methodological and communications 
j improvements will lead, to success. These 
contrasting viewpoints are illustrated by 
' \tWo recent national studies, Rand (Pincus, 
1980^ representing the political reality 
and Northwestern University (Boruch and 
Cordray, 1980) the ^technical. 

- closely related to these is the notion of 
some methodological grail \ -which, when 

* 'discovered, will lead to an educational 
Utopia. This grail is generally assumect , 
to be some form of advanced social science 
research methodology. * p 

■ . ' ' • 

- there is also concern over the impact of 
evaluation studies pn policy and decision 
makers, a feeling that it is too low, and 
sh0ui4.be higher. / This overlooks *the ^ 
threat of change that serious evaluation 
poses to program managers and policy 
makers alike. These people often have 
enough difficulty ^fetting agreement on 

ipolitical grounds, and do not particularly 
ppant to have their investment in compromise 
and negotiation put at-risk by consideration 



of new information from retrospective 
evaluations. These are .consequently 
treated as pro fp^ma reports, and should 
"not be expected to have impact. 

In reviewing the literature provided by NWREL the 

- scope of evaluation i there" appears to be 
some confusion over the scope of evaluation 
as the term is used in the literature, e.g., . 
does it really include testing or information 
systems, or are these lesser activities 
reluctantly carried out by evaluators, whose 
primary calling is research} 

" t 

- autonomy/organizational role. : what can b^ 
inferred from the writings -of some evaluators 
especially in relation to limited impact on 
policy, is that 1}hey conceive themselves as 
having a. certain professional autonomy to do 
research on their own initiative in order to 
provide input and guidance (even direction) 
to 1 policy makers? 

-. rates of change i a factor additional to the 

political nature of* the* evaluator's enviro 
_ _ment is the rapid rate of change in policies, 
program goals, guidelines, social expect- 
ations, etc., not to mention staff and 
< organizational change. These changes 
occur with a rapidity that approaches and 
sometimes is fasters than the timescale 
required to carry otit rigorous evaluations . 

. A s 
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In the limit, this time scale factor can 
make evaluations almost irrelevant to 
decisions? and 

rohustneas of evaluations t as ^ these factors m 
are taken into account, together with the 
presence of a gooddeal ot environmental 
noise,; that is, extraneous factors of 
various kinds, and the often adversarial 
nature o^'pdlicy ma&ing discussions, it is 
a : matter of question whSmer* or not it is 
in fact possible to do robust evaluations 
.today, that is, evaluations* whose results 
cannot he successfully - challtffiged . If this 
is the case, it should give evaluators 
some serious concern* 
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The aim of this section is to describe and discuss 
some ; metaphors that may prove helpful in understanding 
present evaluation practice, and. in supporting 
constructive developments in the future. These 
^^ifietaphors relate to evaluation at two levels: 

- supply metaphors , relating to evaluation as 
- * a supplier of services; and 

v 

* * 

- demand metaphors , relating to the SEA and 

a' 

evaluation unit's clients, who play the 
► role of customers for the unit's services.- 

In considering^metaphors from management, the very 
real qji^stxon of differences between business and 
^government arises • One treatment of management that 
tries to provide a jingle perspective for viewing 
these two contexts is Buchqle, {1977) • # This is also 
of interest because .of his -"strong advocacy of systems 
management, with its emphasis on feedback and infor- 
mation, which is qui.te relevant to the SEA context. 
However, SEA policy making is quite unlike business 
, strategic planning, due?, to the rapid" change possible, 
. in finding and goals imposed by external forces, e.g., 
.'the state legislature. 

Two readable writers on" the systems approach are 
vvChurchman (1968), who gives a balanced treatment of 
thX>sy%teifls approach that is "aware of the social 
complexity of the real world, and Boguslaw (1965), '* 
who criticizes some of the Utopian tendencies of the 
ear^y systems analysis movement.. 
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B.l Supply metaphors ■ * . 
, , l« 

As remarked in the preceding section, evaluation 
is not a monolithic activity, but has a number of 
distinct elements in practice (even if "pure" 
evaluation is only concerned with the first), namely: 

- research analyses; 

- information systems; 

- testing} and 

- expert or technical assistance, which can 
ba associated with any of these. 

A 

B.l.l Metaphors for research analyses * 

* 4 * 

Research analyses are carried out by specialist units 
of varibus kinds- within organizations .r" These include 
a whole family of evaluation methodologies (Swanson, 
1975) i some of which can be carried out externally, 
e.g., by a consultant for a cliefttl t! Ih fact, a. 
^number of these are the kinds of -analyses" carried 
out by internal consulting* units in organizations, 
e.g., operations research, management audit, while 
^ others are associated with budgeting activities^, e.g., 
cost-benefit analysis and programming, planning and 
budgeting (PPBS), and .yet others 'with product 
optimizing activities , e.g., value analysis/engineering 
Of, these, operations research has already been treated 
in the metaphor series of reports (Page, 1979), and 
; value analysis/engineering is remiifLscervt of the 
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special thfrust of formative evaluation. 

Research* analysts^are also employed by public organ- 
izations in a variety 6f other fields, e.g., budget 
analysis, policy analysis^ Another example of a 
research anaUyst is .the library or information 
analyst. Market research- Analysts may conduct 1 
original research analyses or routine surveys, 
investigative journalists also function in a .similar 
way without the organizational background, see, for 
example, the metaphor series of reports (Guba, 1979). 
These research analysts are* characterized by 
specialized training and skills, theirs is technical 
work,^not managerial, or administrative. _ They carry 
out ad hoc studies rather than routine activities, 
and are concerned with^ questions of methodology. 

The most' suggestive metaphors are those of operations 
research/management, science (OR/MS) and marketing/ 
market research. These are discussed at'greater 
length below. 

(a) . Discussion 6t the OR/MS metaphor . - OR/MS, as well 
as more general^forms of consulting, have special 
requirements iikj>erforming services for management. 
These include the negotiation of workable spec- 
ifications f6r tasks, good communications (both in 
mutually consistent views of what is being asked 
fox* by management as well as in clear, timely 
Reporting), and mutual education. Management 9 
should be kept abreast of the kinds of realistic 
methodological capabilities that OR/MS has to 
deliver results with, and OR/MS should understand 
what the organization's goals are and how they 
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are being pursued. When it commits 
resources to tasks, OR/MS is responsive to 
management's sponsoring authorization. OR/MS 
staff should not indulge in private projects, 
nor believe 'that their 'technical expertise in 
some way makes them superior to general management. 
Nevertheless, an explicit awareness of this support 
service role need no t t rob OR/MS staff of high 
professional aspirations and conduct. 

From what the consultant has been able to learn^ 
of evaluation, it seems that good evaluation 
should shar^jnany of the characteristics of* good 
OR/BJS practice in ifoe .area -of dealing with clients. 
In general, seeking good communicatioji with clients 
seems to be acknowledged as a goal, although one 
that is not always achieved in practice. However, 
there is some (implicit) reference in the liter-* 
ature to the professional autonomy o'f the evaluator 
in initiating studies, r which violates the essential 
support role. It is not surprising that such 
studies lack' impact . " 

It takfes time, both staff time and elapsed time, 
to carry out analyses. t If requirements or 
activities are changing rapidly> then long 
running projects cannot be undertaken. If the 
environment is noisy, that is, there are many 
extraneous factors, then methodologies that rely 
upon exact data will not work. The OR/MS 
professional responds to limitations , inherent 
in the context of- the study by selecting an 
economic and appropriate methodology capable of 
giving results *on tinje . If none appear to be 
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available, he advises management, 1>ut does not 
necessarily expect the organization to stop 
functioning until one is developed, or for long 
enough for one^to be applied. 

In some evaluation literature, in contrast, there 
appears to be the assumption that methodology comes 
first, and that policymakers should only propose 
programs that existing methodology can_analyse. 
This is hot a position that SEA evaluators can ' 
afford, even if appropriate to academe. On the 
other hand, there is some truth to this position, 
a variant of "do not promise what you cannot 
deliver" . In the case of the SEA program manager 
preparing his budget, the caution is ."do riot 
promise what your evaluation uni^jannot docuifient" • 

*In the same, direction, a mature field of applied 
professional skills, such" as OR/MS, knows that 
•the gftlf slowiy widens between academe and the real 
world. The academic view of the subject is pure, 
exact, permitting sophisticated methodologies in 
simplified and abstracted settings/ The real 
worJ4 is pragmatic, oriented towards useful results 
rather than theoretical purtxy, and . constrained by 
time and cost. In the same way,, professional v 
evaluators do not do the same kind' of work as 
academic evaluators, and should not regard the 
writings of the latter as prescriptive. On the 
other hand, the tatter should not feel slighted 
by this, since each has their own legitimate^ field 
of endeavor. Ideally feelings of mutual respect 
should obtain .between academics and practitioners, 
but, from experience in other fields, this is easier 
said than realised. 



;(b) Discussion of * the marketing/market research metaphor 
1 In contrast to OR/MS, marketing functions are more 
closely integrated into an organization's mainstream 
and central planning. Marketing s<af:f work 
closely with production and finance to' assure 
that the right amount of the right products are 
produced at the right time and the right cost. 
They are concerned with the consumption of existing 
services as well as planning for new tones through 
needs assessment. They carry out comparative 
studies to find out ways of improving products 
and services. Their situation differs from 
evaluation, however, in the existence of reverilie 
in the business environment as a concrete measure 
of worth . 

The concept of market segementation seems a key 
one for evaluators and educators generally. 
How does one decide to split up student populations 
into distinct groups to be served by different ■ 
programs?, How does program effectiveness depend 
upon the. number of segments? 

Looking at the central role of marketing, e)/aluators 
may be able to learn from the way i» which marketing 
inputs are widely used in organizational decision ' 
making, and how they are able, to deliver effective 
inputs from the 'same messy environment, within the \ 
disciplined* schedule of the organizational- planhing 
cycle . 



B.1.2 Metaphors for information systems 

Data processing and management information systems 
(DP/MIS) are now integrally part of any modern 
organization. DP/MIS is, like educational. evaluation, 
a relatively youi>g activity, with first practical 
applications of co&puters dating from the 'mid 1950s, 
and the MIS movement from the mid 1960s* MIS is 
characterized by the ability to develop, nfelntain 
and exploit a database of information relevant to 
the organization's activities and responsibilities. 
Over the years the emphasis in DP £ias moved away from 

^itd^hoc projects with short life 'to the systematic 
development of long running applications, based- upon 

% a growing database of machine readable information. 

Mathis (1976), Rasp (1980) and Cronin (1980) all 
describe how evaluators deploy such systems,- within 
their own organizational contexts, 

DP" capacity planning is part of any serious planning 
^at the ■ organizational leveTT, since it is now .clear 
i;o all concerned that all applications take machine 
and staff resources, and future applications must 
be planned for. ^ Of course, if the organization's 
requirements change faster than the typical applications 
development cycle, then DP can no longer give current 
- support to the organization and there will be a degree 
of waste in the applications that become obsolete 
before the investment in their development can bq 
4 recovered. 

DP/MIS departments haVe^§ome role in educating their 
organization as to what tnej^current technology can 
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achieve, i.e., what is possible. Ho>^e^, they do 
not indulge in, independent research (except in the 
very largest and then budgeted in advance), biit 
develop specific applications for specific clients 
after careful cost- justification* 

Because DP/MIS is a new organizational function, there 
is an awareness that there is a degree of development 
associated with-it as a field, as well as within any 
particular organization. This has been described by , 
a stage model (Strassman, 1976) in Which four distinct 
stages are identified: ■ 

- initiation * characterized by interest and 
* excitement in the novelty of DP; 

- expansion , characterized by demonstration 

~ - of applications in a variety of organizational 
settings? 

- i 

- control « characterized by a desire to manage 

a variety of DP projects more^ efficiently, 
a narrow, bottom -line approach; and 

- maturity , characterized by the management ~ 
of DP as an effective contributor to the 
organization's purpose's, a* key resource 
component for long range planning, and a 
broader bottom-line approach. 

^< 

The longer DB/MXS has been active in an organization, 
the greater will be the volume of applications software 
and databases. In turn, the maintenance requirements 
will rise, competing for scarce DP/MIS staff., time that 
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can be applied to new developments . - • Thus , long term 
DP/MIS activities, are accompanied by long term 
maintenance "drag", a phenomenon that is receiving 
considerable attention in the computing world . 

c 

More recently, as an outgrowth from the MIS movement, 
there has been the desire to develop better aids to • 
decision-making than MIS typically provide. This 
effort is known by the name "decision support systems", 
and is aimed at taking what is known in advance about 
the structure of the decision, and developing computer 
models working with relevant databases to provide a 
more intimate and powerful support than is possible 
with a general purpose database and an enquiry 
'language. This represents an interesting move to 
narrow the gap between the ad hoc and the routine 
information system, .,and clearly may represent an idea 
df relevance to ' evaluation. Decision support systems 
are well described by McCosh and Scott-Morton (1978). 

Discussion of the DP/MIS metaphor . DP/MES is only a 
little older than SEA evaluation. Its role within 
organizations during; this time has changed, a great 
deal, and this change' has been described by the 
stage model above.. It seems likely that SEA evaluation 
at leist that element that is concerned with, information 
systems, will pass through some similar progression of 
stages. There is an inherent, experience -based 
learning curve", compounded of both organizational „ * 
and -technical factors, whiGh jprobably cannot be 
speeded up. This appreciation should make us more 
patient- when considering the -effectiveness of evaluation 
within SEAs. Time will bring increased maturity. 
Something of this may be already seen in the evolution 
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of the scope of evaluation units in some SEAs. 

» *' r 

A major Issue in many organization^ has been the 
relationship between OR/MS and DP/MIS. In tiie early 
days, OR/MS tended to havp<the lead, because the^ 
were there first, and had the technical Skills tQ S 
operate the qarly computers. Progressively, howevei 
DP/MIS has become the dominant service in mos$ 
organizations. ' Reasons for this include~such things 

as the highly visible investment in hardware, the * \*> 

ft 

desire to use prepared information to answer urgent *j 
questions rather than ad hoc studies, and the initial 
capture of data in machine readable form from the 
. \ operational environment, rather than having to pay' 
* extra for its later entry into the management system. 

. For reasons such as these, it seems likely to the 
consultant that the information^ systems component q& , 
j tha evaluation function Wi^l come to dominate the 

| " analysis and survey components\ * This may lead to a 

I forced reappraisal of individuals^ career goals and ^ * 

■ / their relative importance to the £>EA. , - & 

j The 'leadership of the DP/MIS function in organizations 

j » **%s Another issue of interest by analogy to evaluation. * 

/ There is some controversy over whether or not this 

should be?, a person of general management orientation, 
or a technical expert. Technical expertise is 
increasingly *iess* important . \ More important is an 
understanding of J the goals of the organization and 
. the ability to deliver services to- support management t 
in reaching these goals . This may suggest that the 
leadership of SEA evaluation units (larger ones, ^that 
is, since smaller ones may need all the tecHniq&l 
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expertise that they can get} will pass into the hands 
4 of general managers, especially* in those cases where 
infctmatign .systems become dominant as foreseen above*. 
m '^-Pne factor which may 'work against this move is the 
^ fact that the top echelon of management in an SEA i§ 
either elected or ^pointed, and ''may not have skills 
of a general management nature adequate "to taking 
f * ■ over arid running, an evalu^Qpn unit without its own 
traditional technical leadership./- 

DP/tftfS <|s also committed to systematic capability pf 
• resource planning, integrated with other planning 
^ activities §£ tfie* organization, both in the short arid 
..long- term. This grows out of the increasingly ' * . 
' indispenrs^le role that it plays in orjga^izational 
effectiveness. It seems likely th^t some conjjJonents 
* . of SEA evaluation will come .'to practise similar 
planning as they become more intimately part of 
effet:#Lve SgA operations t * and are perceived as such. 

; I ' " 

B.1.3 Metaphors for testing t ' ^ 

M ^ %{ A number of different disciplines engage in surveys. 
^ These include public opinipn polling specialists , 
and market researchers.' These correspond roughly to 
the testing activity that is often part of Evaluation. 

Thpdk activities are characterized b^ a degree of 
routine application of some enquiry tool, e.g., a/ 
questionnaire, statiJfecal questions of sampling 
size, population homogeneity, ^ reliability, confidence 
limits, and so on., Because of the degree of routine, 
results majnsbe adcorded a significant role in decisiqgi 
* < 
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making in advance, s^nce there is prior 4 expedience of t 

the form of the results and their*past utility. 

* *> 

i * 

. ■ 4 

Discussion of the survey .metaphor .a* The main point 
-that arises from this metaphor is that ^pince surveys 
are not exciting, st^te of the art activities 
general, but rather routine in nature, a premium will 
be placed on value for money, i.e., efficiency. 



B.l.^ Metaphor for expert assistance 

The expert or technical assistance component of SEA ■ 
/ £ ^valuation activity is a form of consulting. This 
« can range over a wide variety of subjects and levels 
of effort.* s Consulting is therefore a metaphor for 
ekpert assistance, but the two are almost synonomous, - 
and so this metaphor does not really extend our under- 
standing of evaluation. It will not be discussed ' 
9 r further. 

B, • ^-J^mand Metaphors 

The two main metaphors here have to do with the 
variety of managerial decision making on the one hand, 
and the necessity of information feedback from the ^ 
field on- the ether. Although closely related, they 
have^differerlfe emphases. 



s 

4. 
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^ B,2.1 Decision making categories \ 

Anthony (1965) .introduced the classic distinction of 
• management^ decisions as falling into the categories of* 

' 9 

* - operational control j 

° 

- management control; and 

- - strategic planning. 

Lucas et al (197^) discuss this framework along with 
related approaches of Simon and Scott Morton. Simon 

«■" "7" t 

uses the terms intelligence, design, choice to .describe 
decisionmaking at the, broadest level, and Scott .MortolT) 
emphasizes the degree of structure in a decision, 
speaking of structured, semi -structured and unstructured 
decisions . The information systems that support each 
of these classes of decision will be different, and 
s . this is reflected in the literature and practice of 
MIS. . x . 
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In actual fact, these three decision categories are 
riot absolutely distinct, but tend to form a continuum 
in practice. , Anthony describes a number of information 
characteristics for each of the categories. These 
are shown overleaf. 

Evaluation, broadly considered, may support each of' 
these-types of decision making, but not in the same 
way. Further, because of the' SEA setting, strategic 
planning is not such a clearly defined activity as .it is 
in business, because of the difficulty of developing 
and applying coherer^ strategy in an extremely 

* * 
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Information requirements for decisions ins 



( 



Operational 
. control ' 



1 • yery detailed 
da*a^ 

r 

2. related to a 
specific task 



3« frequently 
reported 



Management 
control 



moderately 
detailed data 

related to 
achievement of 
organization 1 s 

regularly 
reported 



Strategic % J 
planning 



„■ aggregate 
data 

^ 

related to 
establishing . 
broad policies 

infrequently 
reported 



\ 



*K historical- 
data 



historical and 
predictive data 



predictive data 



5» internally 
generated 



mostly 

internally 

generated 



externally 
generated 



6* very accurate 



7. repetitive 

8 * often non- 
financial 



accurate within 
decision bounds 

exception 
reporting 

mainly 
financial 



accurate in 
magnitude only 

unique to problem 
batfng considered 

often non- 
financial 



Th^e Anthony Framework 
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changeable environment. Additionally, operational 
control is more likely to be practised at the local 
district level than in the SEA. Failure to 
distinguish between these different requirements is 
likely to lead to confusion. 

Kling (1980) and Keen (1981) both treat the organ- 
izational context of information systems in a ma nner — 
that extends Anthony's rather general categories 
to more realistic political environments, such as are 
found in SEAs. Research shows, for example, that 
much human information processing tends to be simple, 
experiential and non-analytic, yet fairly effective. 
The information 'dependencies of SEA management are real, 
but not so direct or determining as is sometimes 
suggested. , 



B.2.2 Feedback Control 

» 

^A simple abstract model of information and decision 
making has been proposed by Yovits and Ernst (1968), 
and discussed in a number of papers since, e.g., 
Yovits et al (l98la,b). This control -systems model 
is highly abstract, but it does remind us of important 
% features of general systems , • including feedback, time 
constants and noise, all of which are relevant to 
considering the systems character of an SEA, within 
e which the evaluation unit functions. These features 
may go some- way to explaining *Some of the inherent 
limitations on what evaluation is able to achieve in a ' 
practical situation. 'For example, if feedback* is not 
employed, or if if' is employed but time donstants or' 
noise are unfavorable, then the organization cannot 

* v 
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follow an effective course of actioil with any 
confidence * 

Another notion presented by this model is that of a 
transducer, a means of converting information ~from 
one medium into another • In a sense, evaluation ± 
acts as a- transducer, or sensor, in extracting 
information in useful structure from the unstructured 
social environment in which education actually takes 
place . 
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MANAGEMENT CONSULTING' 

One of tl\e aims of the Program in sponsoring the present 
study is to use it as an example of a study carried out 
from^c management consulting perspective. This is in 
addition to the substantive results of the study. 
Therefore it seems useful in this appendix^ to give a 
brief description of management consulting, together 
with some sources for further reading.. This may be 
of interest both to NWREL, , which does a significant 
amount of consulting itself, as well as to evaluators, 
who, as we have seen, also play a, consulting role in 
their own environment. 

Generally, '"management consulting is concerned .with 
specific assistance given to an organization by an 
outside consultant and characterized by extensive 
on-site activity on the part of the consultant, 
although the term is often used more broadly. One " ► 
definition, for example, of classic? management 
consulting reads % \ 

"TheJ service provided -by an independent and 
qualified person or persons in identifying 
and investigating problems^ concerned with 
m policy, organization, procedures and methods? 
• 1 recommending appropriate action and helping 

to implement these recommendations 

Reasons for employing a consultant includes their 
special skills, the need for intensive professional 
help on a temporary basis, or the need for an 
intelligent and impartial outside viewpoint* 
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The present study, therefore, is not a typical 
management review, since it is indirect, based in part 
upon literature, with visits to SEAs for field input 
and checking of hypotheses. The study has been 
sponsored by NWREL, but is concerned with SEAs, who do 
not work for NWREL. This naturally limits their 
participation and interest in the study. If the SEAs 
themselves had commissioned the study, at a senior 
level, then more time would have been spent with them, 
and" the results and recommendations would be directed 
to tMem. However, this would not have been the usual 
form of a multi-client study, which more usually deals 
with market research, technology assessment, industry 

profiles, etc. 

• ! ■ • 

\ 

As consulting has developed, some firms have come to 
specialize, or work in specialized settings, e.g., 
•business or government, a particular sector such as 
forestry or education, a particular management 
function such as marketing or -personnel . Larger 
"> consulting firms have diversified extensively to 
include such services as traditional management 

r 

consulting, -technical services in economics or data 
. processing, ^executive /recruitment^, training and/ 
education, technology ^research, ^applied research 
.studies, policy analysis} and so on* In addition, 
major CPA firms have also diversified into management 
consulting, as have also other service companies, such 
as, for example, \ large computer service . firms . 

Thus management consulting has a degree of breadth and 
diversity in practice which is not always represented 
by 'works on the subject. Many of the hallmarks of 
good management consulting practice are shared by 



other specialized consulting services. Thfe most 
important of these is the relationship between client 
and consultant. This includes the way in which the 
the^ consulting task'is specified or negotiated, and ^ 
the consulting firm '(or individual) selected, the ' 
expectations on each side/ the maintenance and 
development of the relationship through the period of 
service, the focus upon mutually agreed results, the 
timely delivery of agreed products (considered in the 
widest sense), ar^jl so on. 

Within the framework established^^ a client paying 
real money for a real "service-,* i.e., the client ^ 
wants certain results or outcomes for his ^ organization, 
features of management consulting that are distinctive 
include t 

- flexibility of means to achieve ends, i.e., 
the consultant is pragmatic and may select 
one of a number of different approaches as 
appropriate ♦ There is no a priori limit- 
ation to a specific technique, as in the f 
case of hiring a specialized consulting 
service* 

. - .related to this, -the consultant may bring* 
in additional technical expertise as 
required (with the prior agreement of -the 
client), serving as a kind of knowledge 
broker between such resources and the 
client i 

9 
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the emphasis upon b^iefits for the' client, 
and' with the objectives of the client 
organization (just what is it really trying 
to do?), rather than solely at the unit 
lev§l, i.e., a holistic approach to, the 
client j ' - , * 

a diagnostic phase of learning, listening, 
thinking, prior to developing specific 
approaches to achieving solutions; 

related to this, the awareness that the 
expressed needs of .the client may not be. 
the real needs, and the ability to uncover 
the latter and work towards them within the 
framework of trust already established; 

a desire to produce the maximal beneficial 
change for^the client, which leads to the 
advocacy of recommendations that are 
realistic and achievable rather than idealized 
and . unachievable ; 

the communication of results through reports, 
discussion, presentation, with simplicity, 
clarity, and emphasis upon key elements, , 
#ith the sole criterion* is it clear to the 
client* 

a framework which explicitly includes the 
period after service, the client's world 
downstream of the final report, when the 
changes recommended are likely to come about 
and have their effects; and 
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- a willingness to continue service into the 
implementation period should the client 
wish • 

Although these are some of the features that character- 
ize the central practice of management .consulting, 
i.e., advising on, or working with a client to produce, 
beneficial change, most can also be identified in 
the other types of services that management consulting 
firms undertake for clients , such as training, or 
applied research. 

Despite the importance of management cdnsulting in 
supporting the. practice *of business and^overnment , 
and often the leading edge of these, it does not have 
a correspondingly high level of literature . Much 1 of 
this is due to the confidential nature of consulting 
assignments, so *that they do not contribute to the 
open literature. On the other hand, part may also 
be due to tfr^ "fact that academic faculty are often in 
competition as consultants in the outside world, and 
are somewhat scornful df the pragmatic and eclectic 
approaches employed in the real world. Also, 
the client-centered approach of the professional 
consultant is at variance with the theoretical approach 
of the faculty member, whose i^igi3J?^allegiance is 
often, through necessity, to the advancement of 
knowledge through publication, rather than to client 
well-being. ' * 

Having said this, some useful sources include t 

- Bell, Chip R, and Nadler, Leonard (eds) 

The Client -Consultant Handbook , Houston, TX, 
Gulf Publishing Co., 1979. 
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- Fuchs, Jerome H. Making the Most 'of 
Management, Consulting Services , New York, 
AMACOM (Division of American Management' 
Associations),, 1975. 

* 

- Hunt, Alfred. The Management Consultant . 

t- ■ 

New York, Ronald Press, 1977. 

- Kubr, M (ed). Management Consulting: a 
Guide to the Profession . Geneva, International 
Labour Office, 1977. 

Trade associations and professional. societies in, the 
US include: 



^Association of Consulting Management 
Engineers, Inc. (ACME) * 

- Association, of Management Consultants 

- Institute of Management Consultants 

v 

e 

- Society-of Professional Management 
Consultants , Inc . 
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SEAS VISITED' > 

Visits were carried out to three SEAs, coordinated 
by the directors of the evaluation units: 

California. Dr Alex Law 

Montana Dr William Connett 

Washington Dr Alfred Rasp, Jr 

Tffe consultant is grateful to these directors and ' 
their SEA colleagues for making their time so 
generously available for interview and discussion. 
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